Un-blind the link checker, tighten the ratchet, and require breaking new tests - #574
Merged
Conversation
The fault-injection audit (20.11) planted a broken link and watched it pass. Cause: production renders internal links absolute and lychee --offline excludes every http(s) URI, so 133,874 of 149,516 links were excluded and the job was green because it inspected almost nothing - the homepage's single OK was its own skip-link anchor. --remap resolves them against the built tree: 15,642 -> 114,050 links actually checked. Turning it on found five real defects nobody could see: - /blog/2025/complete-guide-... x2: the link carried the directory, the slug drops it - /blog/rails-8-1-active-job-continuations-END-LOST-... x3: link used the directory name, the real slug omits it - and the post's OWN canonical_url pointed at that 404 - /contact/ x2 (a conversion path) should be /contact-us/ - /resources/internal-product-roi-calculator: never existed. The section promised a spreadsheet and itemised five things inside it, no-email instant-download. There is no /resources/ section; the sibling lead magnet shows the real pattern (content/lead-magnets/ + download_file). A promise with nothing behind it is the same class as an unsourced number, so it is removed - the post already closed with a working consultation CTA. Same audit: the rendered banned-phrase ratchet sat at 14 against an actual 11, and those three spare hits swallowed a planted banned adjective whole. Tightened to 11 and proven exact (fails at 10). SURFACES never globbed content/next/**, so the v2 rail had no source-side cover at all. Verified: links 0 errors / 114,050 OK; unit 287 runs, 6143 assertions, 0 failures; hugo-build clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
Paul, 2026-08-22. Green proves a test RUNS; it does not prove it WORKS, and the two look identical in CI. The rule now says inject the defect the test exists to catch, quote the failure, then revert and re-run green - both halves are the evidence. Grounded in what the fault-injection audit measured the same day: the testimonial gate would have passed a vanished blockquote (every string contains ""), the rendered ratchet had 3 hits of slack that swallowed a planted phrase, and the link job excluded 90% of links while reporting green for a year. Also states the honest converse: when a gate cannot discriminate yet (derived tenure, blind until 2027-01-01), write that in the test rather than letting a pass read as proof. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-through on the fault-injection audit (#572 / 20.11), which caught 3 of 8 planted defects.
The link checker was inspecting almost nothing
Production renders internal links absolute;
lychee --offlineexcludes every http(s) URI by design. 133,874 of 149,516 links were excluded — the homepage's single "OK" was its own#main-contentskip-link anchor. It had been green for a year.--remaponto the built tree takes it to 114,050 links checked.Turning it on found five real defects nobody could see:
/blog/2025/complete-guide-...×2/blog/rails-8-1-...-end-lost-...×3canonical_url/contact/×2/contact-us//resources/internal-product-roi-calculatorThe last one wasn't a broken link. The section promised an "Internal Product ROI Calculator" spreadsheet and itemised five things inside it, "no email required, instant download". There is no
/resources/section, and the sibling lead magnet shows the real pattern (content/lead-magnets/+download_file). A promise with nothing behind it is the same class as an unsourced number, so it's removed — the post already closed with a working consultation CTA. If the calculator exists somewhere, say so and it comes back as a real lead magnet.Two ratchet gaps
RENDERED_BASELINEsat at 14 against an actual 11 — those three spare hits swallowed a planted banned phrase whole. Tightened to 11 and proven exact (fails at 10, passes at 11).SURFACESnever globbedcontent/next/**, so the entire v2 rail had no source-side cover.New standing rule
A new test isn't done until you've broken the code and watched it fail. Green proves a test runs, not that it works — and the two are indistinguishable in CI. Inject the defect, quote the failure, revert, re-run green. Both halves are the evidence. Where a gate genuinely can't discriminate yet (derived tenure is blind until 2027-01-01), say so in the test body rather than letting a pass read as proof. In
CLAUDE.md§TDD and.okf/build/test-gates.md.Verified
rake test:links0 errors / 114,050 OK · unit 287 runs, 6143 assertions, 0 failures ·bin/hugo-buildclean. Content-only changes otherwise, so the visual suites don't apply.🤖 Generated with Claude Code
https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ